ZeroBERTo: Leveraging Zero-Shot Text Classification by Topic Modeling

Traditional text classification approaches often require a good amount of labeled data, which is difficult to obtain, especially in restricted domains or less widespread languages. This lack data has led the rise low-resource methods, that assume low availability natural language processing. Among them, zero-shot learning stands out, consists classifier without any previously data. The best results reported with this approach use models such as Transformers, but fall into two problems: high execution time and inability handle long texts input. paper proposes new model, ZeroBERTo, leverages an unsupervised clustering step obtain compressed representation before task. We show ZeroBERTo better performance for inputs shorter time, outperforming XLM-R by about 12% F1 score FolhaUOL dataset.